1. Field of the Invention
The present invention relates generally to a method for eliminating the bottleneck created in a large interactive computer system environment when jobs are submitted to a component of an operating system faster than the component can receive them. It relates specifically to an improved internal reader function of the operating system for dynamically and efficiently processing job-created input streams.
2. Description of Prior Art
An internal reader (INTRDR) is a subsystem function of the IBM Multiple Virtual Storage/Extended Architecture (MVS/XA) operating system. In particular an internal reader is a function of the job entry subsystem (JES3) of an interactive computer system.
MVS/XA--managed environment: The computer systems that MVS/XA manages such as models of the IBM 3090 complex are capable of multiprogramming, i.e. executing many programs concurrently. For example, hundreds of jobs can be executed simultaneously for time-sharing users who could be at distant geographical locations. The computer systems that MVS/XA manages are also capable of multiprocessing, i.e. simultaneous operation of two or more processors that share the various system hardware devices. MVS/XA provides each user with a unique address space (virtual storage addresses) and maintains the distinction between the code and data belonging to each address space.
MVS/XA organizes each job in a system as tasks and attempts to process each one. The tasks for one job generally compete with one another, and with tasks related to other jobs, for use of system resources. A component of the MVS/XA operating system, the supervisor, controls the progress of the tasks through the system. The supervisor allocates resources (other than input/output (I/0) devices) and maintains current information about each task so that processing can resume from the appropriate point in case the task is interrupted.
Modules of the MVS/XA operating system normally store the information needed to control a particular task or manage a resource in defined areas called control blocks. Control blocks representing many tasks of the same type may be chained together on queues, i.e. each control block contains the starting address of (points to) the next one in the chain. A program can then search the queue to find the data for a particular task or resource which might be: an address of a control block or required routine, actual data (value, parameter or name) and/or status flags. All fields in a control block are predefined.
Communication between MVS/XA components or subsystems (programs) occurs due to the use of common macro instructions. These instructions invoke segments of program code that map frequently used control blocks or perform frequently used system functions. Macros exist for such functions as opening and closing data files, loading and deleting programs, and sending messages to the system operator.
The job entry subsystem (JES3): A subsystem, such as JES3 12, communicates with MVS/XA through a subsystem component known as the subsystem interface (SSI) 14 as shown in FIG. 1a. MVS/XA manages each job (a JCL (Job Control Language) statement in an input stream) during execution while the job entry subsystem of the operating system manages each job before and after execution. In this case, an MVS/XA operating system requires the JES3 in order to process each job. JES3 reads an input stream (a collection of jobs) which are identified by a JCL statement and performs work to prepare those jobs for execution. That work can be related to a specific job from the input stream, or it can be related to work JES3 must perform in its role of coordinating the preparation of many jobs for MVS/XA execution. There may be special kinds of JES3 jobs where no JCL is involved. JES3 reads each job in the input stream and places it on at least one direct access storage device (DASD) also known as a spool device. A spool device serves as a buffer between input devices and routines that read input data, and between routines that write output data and output devices; storage place for the control blocks and data JES3 builds to process jobs; and, a collection point for data to be distributed to individual processors in a multiprocessing environment. SPOOLing is the temporary storing of jobs and job-related data in intermediate stages of processing on the spool device so that they are readily accessible. Since each job has a job class, priority, and output class, JES3 selects jobs from the spool device for execution in a way that encourages the efficient use of system resources.
JES3 processing occurs in six stages: entry, conversion/interpretation, device allocation, scheduling a job for execution, output and purge. On entry, JES3 reads a collection of jobs from an input/output device such as a card reader, remote terminal, another MVS or MVS/XA system, tape drive or DASD. Each job itself could create input streams. Job-created input streams are processed by a JES3 internal reader program. In general, the internal reader transfers an input stream from an output service, i.e. an output queue, to the JES3 input service, i.e. an input queue, to be processed. That is, any job executing in MVS/XA can use an internal reader to pass an input stream created by the job to JES3, and JES3 can receive multiple job-created input streams concurrently through multiple internal readers. The job-created input stream is not entered from a device such as a card reader, remote terminal or direct access device. Instead, the input stream is created by a job as an output data set, e.g. a data set containing job control language (JCL) and/or data and/or JES3 control statements and comments.
JCL is a special batch language which is used to create MVS/XA data sets. A special JCL statement is used to allocate a data set to the JES3 internal reader. In other words, a job creates an output data set and requests that it be processed as a job by coding the destination as SYSOUT=(*, INTRDR) in the data set identifying (DD) statement. The statement has the form //ddname DD SYSOUT =(*,INTRDR) in which "ddname" is the name the batch program will use to open, access, and close the data set. "DD " is used to allocate a new or already existing data set. "SYSOUT" identifies this data set as a system output data set, i.e. a SYSOUT data set. "INTRDR" indicates to JES3 that the SYSOUT data set is to be sent to the JES3 internal reader as an input job stream. (SYSOUT is a data set that is managed by JES3 and is discussed below.) This eliminates the need to reenter job-created input streams in the conventional manner using an I/0 device. In general, the JES3 internal reader program transfers a job-produced input stream (an output data set such as SYSOUT) directly to JES3 input service programs that normally process jobs obtained from I/0 devices. As JES3 reads the input stream, it assigns a unique job ID to each job (in the input stream) and places each job's JCL, optional JES3 control statements, and input data into spool data sets. The input stream jobs are then selected from the spooled data sets for processing and subsequent execution.
Batch jobs are selected by JES3 in response to requests for work from the initiator function of the MVS/XA job scheduler. They run in the initiator's address space. (An initiator is an MVS/XA system program that is started by either the operator or JES3 in order to request, and subsequently execute, batch jobs submitted via JES3.) Jobs created by TSO LOGON, the MOUNT command, or the START command (discussed below) are selected for processing when they are entered by a process known as "demand select". These jobs run in their own address spaces.
In the C/I (Conversion/Interpretation) stage, JES3 uses a converter program to analyze each (input stream) job's JCL statements. The converter takes the JCL and converts it into internal text that is recognized by JES3 and the job scheduler functions of MVS/XA. JES3 then calls the interpreter function to further analyze the JCL and build control blocks. Both the internal text and control blocks are then stored in the spool data set.
A job generally needs to use I/0 devices, such as tapes or DASDs and data sets when it runs. MVS/XA assigns these resources to jobs through a function called "device allocation". Device allocation uses the information in the job's JCL statement to assign the proper resources to the job. The job is then passed to an MVS/XA initiator.
The scheduling phase of JES3 responds to requests for jobs from the MVS/XA job initiator function. JES3 will select a job from a job queue on a spool data set and make it available for the initiator.
JES3 controls all system output (SYSOUT) processing. SYSOUT data is produced by jobs during execution. While running, a job can produce system messages that must be printed, as well as data sets that must be printed or punched. After completion of the job, JES3 analyzes the characteristics of the job's output in terms of its output class and setup requirements and processes its output accordingly. Specifically, JES3 gathers the output data by output class, device availability and process mode, then queues it in the SYSOUT data set temporarily on the spool device for output processing. That is, MVS/XA creates a temporary SYSOUT data set on the spool device to contain the records written to the SYSOUT data set. (After job execution, JES3 prints or punches the SYSOUT data to the appropriate output device. The printers and punches are JES3 devices, identified as such during system initialization.)
At the purge stage, when all processing for a job is completed, JES3 releases the spool space assigned to the job, making it available for allocation to subsequent jobs.
JES3 input service programs: JES3 programs, called dynamic support programs (DSPs), perform the work JES3 is required to do to prepare jobs located in the input stream for execution. There are three types of DSPs:
1. Resident DSPs that are a fixed part of JES3 processing. PA1 2. DSPs that are invoked by an operator command (* CALL DSP name). PA1 3. DSPs that process units (or items) of work required by a job. PA1 1. all internal reader data sets; PA1 2. spinoff data sets created by JES3 DSPs; and PA1 3. jobs submitted from other systems via a systems network architecture/network job entry network. PA1 1. As more jobs are added to JOBO, its JDS increases proportionally in size by the addition of chained (SRF) records. This results in longer search times, i.e. search of the chained records, as the number of input/output (I/0) calls increase because each SRF record must be read from disk storage and, if changed, written back to disk storage. The program which processes the long JDS chain is also responsible for processing TSO LOGONs. PA1 2. The module IATMSGC controls the processing of the JOBO JDS (via calls to another module (IATDMJA)). However, module IATMSGC is also directly involved in processing demand select requests, e.g. TSO LOGON, the MOUNT command and the START command. Since module IATMSGC is a DSP driver with a DSP maximum use count of one, it can only process one request (job) at a time. This results in competition for processing time between internal reader jobs and demand select requests. This, of course, creates excessively long times for TSO users to log on to the system. If module IATDMJA is spending an inordinate amount of time on I/0 while processing internal reader jobs, demand select jobs will be delayed, or even stopped, if the JOBO JDS is in use. (Even without the JOBO internal reader support, performance studies have shown that the IATDMJA module is a source of a major I/0 performance bottleneck.)
When JES3 processes a job, the eventual result is the execution of one or more DSPs. In general, DSP 18 as shown in FIG. 1b handles one unit of work and performs one particular function, but its work is accomplished through a pair of JES modules, i.e. a driver module 20 and a data CSECT 22. (The data CSECT contains work areas, constants and common routines used by the driver module and other modules that the driver module invokes. Not all DSPs however, require their own data CSECT.) JES3 uses internal reader dynamic support program (INTRDR DSP) routines to handle the transfer of job-produced input streams to the JE83 input service. A DSP is automatically initiated when a job is created that has scheduler elements. The DSP is, itself, a JES3 task. JES3 provides for its own internal dispatching using the DSP. The operator invokes as many internal reader dynamic support programs, i.e. copies of the INTRDR DSP, as necessary by using the *,INTRDR command in order to handle the current load of JES3 job-created input streams to be transferred to the input service.
In general, all DSPs are dispatched using the JES3 master dispatching queue called the function control table (FCT) chain 24 as shown in FIG. 1b. Every DSP corresponds to one or more FCT entries in the FCT chain as suggested by line 26 "connecting" DSP 18 with FCT entry 24a. JES3 dispatches an FCT entry in very much the same way as MVS/XA dispatches a task control block (TCB). The FCT entry representing a DSP i one element on a chain of elements comprising the FCT chain. The FCT chain elements are arranged from highest to lowest priority according to the priority assigned to the DSP. In general, FCT entries represent JES3 dispatchable units of work for all jobs.
There are two types of FCT entries: resident and non-resident FCTs. Resident FCT entries are not dynamically added or removed from the dispatching queue. These resident FCT entries represent DSPs that perform required JES3 functions like operator communication and output services. Non-resident FCT entries are added and deleted from the dispatching queue as needed. These non-resident FCT entries perform operator utility functions as well as the core processing of JES3.
A JES3 job is defined by a control block called a job control table (JCT) entry 30 as shown in FIG. 1c. The JCT job structure is created by JES3. (An MVS/XA job defined by a JCL JOB statement is analogous to a JES3 job defined by a JCT entry.) Every JCT entry contains the JES3 job to be performed, i.e. the characteristics of the job, and one scheduler element for each unit of work that must be done (in sequence) to process that particular job. As shown in FIG. 1c, a scheduler element represents a stage of the JES3 job to be processed, i.e. the C/I stage (32), the MAIN stage or device allocation stage (34), the SYSOUT output processing stage (36) and the purge stage (38). These scheduler elements, representing units of work, are processed sequentially, and each is represented on the FCT chain by a DSP that performs the work required for that specific scheduler element. In general, the scheduler elements in a JCT entry represent units of work for a specific MVS/XA job. A job segment scheduler (JSS) selects scheduler elements that are ready for processing and builds entries on the FCT chain 24 as shown in FIG. 1d so that DSPs representing those scheduler elements (32-38) will be dispatched to do the work required by those scheduler elements. The JSS scans the job queue (the FCT chain) in priority order, finds a job that has a scheduler element to be processed, builds a resident queue element control block (RSQ) for that scheduler element and associates the resident queue with an FCT entry that corresponds to the DSP that will be dispatched to perform the function required by that scheduler element. For each DSP not represented by a resident FCT entry, an FCT entry is built by the JSS and added to the FCT chain. This "new" FCT entry will also represent a DSP that will do the work requested by a scheduler element.
The JSS schedules work, for JES3, that must be done on behalf of an MVS/XA job which is defined by scheduler elements. (The scheduling done by the JSS corresponds to the dispatching done by the multifunction monitor (MFM).) Pointers from the JCT entry to the job's other control blocks are placed into the RSQ. The RSQ, a large in-storage-only control block, is a storage area for status flags, job information fields and queueing pointers. RSQs last only for the life of a scheduler element.
The MFM, the actual JES3 dispatcher, scans the FCT to dispatch the appropriate FCT to accomplish scheduler element work. When the MFM finds a dispatchable FCT entry, it restores the DSP environment from the FCT entry and then passes control to the DSP driver module for processing. The DSP then performs its service and the MFM relinquishes control until the DSP returns control to it. The MFM then saves the DSP environment (register contents, etc.) in the FCT entry for the next dispatching of the DSP. The MFM then begins to scan the FCT again. The JCT-FCT structures permit JES3 to track and schedule work on a job-by-job basis (JCT entries), and to dispatch work on a unit of work (within a job) basis (FCT entries). The JSS provides the interface between the JCT and the FCT by means of the scheduler elements in the JCT and a DSP (which does the work of a scheduler element represented by at least one FCT entry) on the FCT chain. Sources of JES3 jobs: There are several sources of JE83 jobs: normal jobs which are entered either by the MVS/XA internal reader or by locally attached devices such as card readers and DASDs or by remotely attached devices or by another JES3 node (in a networking environment); "demand select" jobs, which occur when an operator issues an MVS START or MOUNT command, or a time-sharing option (TSO) user enters a LOGON command; and, "called jobs" which are created by operator request in response to a JES3 * CALL command.
Internal reader jobs: A job is generally submitted to the JES3 internal reader (discussed below) in one of two ways. One way is through the use of a CLOSE (macro) to close a SYSOUT data set, e.g. one assigned to the internal reader via SYSOUT=(*,INTRDR). In response to the CLOSE macro, MVS/XA invokes the CLOSE SVC routine which sends internal reader job data set (JDS), i.e. SYSOUT data set, information to JES3 via the subsystem interface (SSI). A staging area (STA) will be created which will include a pointer to the SYSOUT data set, i.e. JDS, entry. The STA is sent to the JES3 address space via the SSI. The other way is when a time sharing option (TSO) user interactively submits a job (demand select jobs) via the TSO SUBMIT command. The format at the SUBMIT command is SUBMIT dsn where dsn is a data set containing a job input stream. As a result of the user-issued SUBMIT command, TSO dynamically creates or allocates an internal reader data set, and writes the job control language (JCL) records to the internal reader via SSI. In particular, JES3 will allocate from 2 to 9 spool track groups to the internal reader data set. The data set will be processed as a JES3 multiple record file (MRF). A resource allocation block (RAB) is created in the TSO user's address space. This block is used by JES3 to process read and write requests in the user's address space and contains pointers to all of the spool track groups allocated to the data set. JES3 will create a job data set (JDS) entry for the internal reader data set. The JDS entry contains a pointer to the first record of the data set on spool. This JDS entry is part of the JDS block corresponding to the internal reader job representing the TSO user. TSO will read in all records from the data set specified by the SUBMIT command and write them back to the internal reader data set. When all of the records have been transferred, TSO will send the internal reader data set to the JES3 address space. A staging area (STA) will be created which will include a pointer to the internal reader data set JDS entry. The STA is sent to the JES3 address space via the SSI. In both cases, the SSI request to process data sets is sent to the JES3 address space. A module (IATMSGC) receives the request and calls another module (IATDMJA) to process the request, i.e. process the STA. The IATDMJA module obtains a job number (AJOBNUM) and then issues a SPINOFF macro. The IATDMJA module calls the module IATOSDR to process the internal reader data set pointed to by the STA as a spinoff data set. That is, module IATOSDR copies the JDS entry of the new input job stream (effectively adding the internal reader data set) to a JOBO JDS block (which is discussed below).
In other words, internally, the output service treats a data set, e.g. a SYSOUT data set, destined for an internal reader as a spin off data set which JES3 can process while the job (that created the data set) is still executing. In doing so, the output service adds an entry (corresponding to the data set) to the JDS block representing a pseudo-job (JOBO) that is created during JES3 initialization. JOBO is strictly used to keep track of spin off data sets created by JES3 DSPs. The output service then builds an output service element (OSE) for the data set and queues the OSE on the output service hold queue. At this point, processing stops until a copy of an internal reader (INTRDR DSP) is dispatched. If there are no free copies of the INTRDR DSP program, nothing happens until the operator invokes (calls) an internal reader using the *,INTRDR command.
JES3 Internal Reader--JOBO
JOBO is a special "pseudo-job" created on a DASD by JES3 during output service initialization. JOBO represents the job stream containing the DD statement. This "pseudo-job" job structure is created with the following control blocks. (These control blocks in relation to the JCT entry are shown in FIG. 1e.)
A job description and accounting block (JDAB) 40 is constructed simultaneously with the JCT entry 30. JDABs provide DSPs with data about jobs and make JCT access unnecessary. In general, entries representing each scheduler element in a JCT are appended to the JDAB to provide DSP accounting for each DSP needed for the job.
A job data set (JDS) 42 control block is constructed during JES3 input service processing and initially contains the data set which contains the JCL for the job (JESJCL), two message data set for system generated messages as well as for messages generated by DSPs (JESMSG), and any SYSIN data for the job (JESIN). Entries for (pointing to the) SYSOUT data sets (discussed previously) are added to the JDS when the data sets are opened during MVS/XA execution. This job data set also contains the records written to the SYSOUT data set.
A job management record (JMR) 44 control block is built during input service processing and contains the job accounting information.
An output service element (OSE) 46 control block is created to contain a set of SYSOUT data set characteristics relating to one or more data sets to be managed during output service processing. The first OSE for a job is constructed during input service processing and written to a spool volume. Later, during output service processing, this OSE is completed and others may be generated for the job. These elements become the "output queue".
A job track allocation table (JBTAT) 48 is created and is a list of spool track groups allocated to a job.
There are other job related control blocks that are not necessary to discuss for an understanding of the implementation of the invention described herein.
Since no JCT is created with JOBO, JOBO is ineligible for scheduling by JSS because there are no scheduler elements. (That is, the JDAB will not have any scheduler elements which JSS can use to build entries in FCT.) JES3 merely uses JOBO to store spinoff data sets, provided by the following, until they can be processed by the output service:
The spinoff data sets that were created by jobs executing in MVS/XA and that are stored on JOBO are submitted directly to JES3 output service using internal reader programs.
Internal Reader Processing: MVS/XA processes data sets allocated to the internal reader by sending them to JES3 via the SSI which provides the communication interface between MVS/XA and one or more subsystems like JES3. In effect, MVS/XA requests services from JES3 by using the SSI. Some requests can be handled entirely by JES3 SSI modules in the caller's address space. However, in most cases, the request needs the services of the JES3 address space. In the JES3 address space, the SSI request is processed by module IATMSGC which, in turn, calls module IATDMJA. IATDMJA submits the data set to spinoff processing via the JES3 SPINOFF macro as explained above.
The SPINOFF macro is processed by module IATOSDR. IATOSDR will read in from the spool device each single record file (SRF), i.e. each entry, of the JDS belonging to JOBO. IATOSDR will stop once it finds a free JDS entry or when it gets to the last SRF in the JDS, i.e. SRF, chain. If a free JDS entry is found, IATOSDR will copy the (internal reader) data set into the JDS entry. Otherwise, a new JDS entry will be created in the last SRF of the chained SRFs and the data set will be copied into the "new" JDS entry.
The JES3 internal reader (INTRDR) is a "called job" (a request) which is not defined by JCL. (This called job is internally generated by JES3 in response to the *CALL command.) The JCT entry corresponding to this called job contains an INTRDR scheduler element, representing the DSP, i.e. INTRDR DSP, needed for this request (to process the data set in a JOBO JDS entry), and a purge scheduler element. When the INTRDR DSP is dispatched, it reads the OSE for the data set, locates the appropriate entry in the JOBO's JDS and calls the JES3 input service to process the data set as an input stream. The driver module for the INTRDR DSP created by JSS is IATISIR. IATISIR calls the normal JES3 input service DSP driver, IATISDV, to process the data set. IATISDV issues the IATXOSPC macro which invokes a JES3 output service routine to scan the JOBO JDS chain for a JDS pointer to an internal reader (spinoff) data set on JOBO. If it finds one, a pointer to the JDS entry on JOBO is passed back to IATISDV, which then loads and processes another module (IATISLG) to process the data set on JOBO as if it came from a physical card reader. Input service then builds the usual job control blocks for the data set, including a new JDS for the job, and releases the JDS entry, corresponding to the original output data set, from JOBO's JDS. When the data set or job stream in the JOBO JDS entry has been processed, i.e. upon return from the IATISLG module, IATISDV will again issue the IATXOSPC macro to find more internal reader work. If another JDS entry is found, control is returned to the IATISLG module to process the data set on JOBO. The above procedure will continue until JES3 output service cannot find any more JOBO JDS entries for the internal reader. That is, control returns to IATISIR which then enters a wait state. IATISIR will stay in this wait state until another internal reader data set is spun off by IATDMJA. In this case, IATOSDR will force IATISIR out of its wait state after it adds the JDS pointer to the data set to a JOBO JDS entry. IATISIR will then call IATISDV and then go into the wait state again following the processing of at least one other internal reader data set. This process continues until the internal reader job is removed from the system.
Internal reader jobs are started and ended manually by system operators. In order to start an internal reader, the system operator issues an *X, INTRDR from a system console. Once issued, control passes to the internal reader driver (IATISIR). (To remove an internal reader job, the system operator would issue an internal reader *C INTRDR from a system console.) This manual process means that the system operator would have to determine (guess) how many of these jobs would be needed to handle a specific work load. If his guess was too low, then there would not be enough internal readers to efficiently handle the work load. If his guess was too high, then the system would have the overhead of maintaining one or more idle internal readers, each one represented by an FCT which would have to be scanned by the MFM to determine that it did not have to be scheduled.
The above process creates a "bottleneck", in a large interactive computer system environment, when jobs (requests) are submitted to the JES3 input service at a rate that is faster than JES3 can receive them Of course, the length of the queue is shortened each time a request is processed, but if requests arrived at a rate faster than they can be processed, the queue can get very long. This results in a very long JOBO JDS chain, causing a backlog in output service select processing because time consuming I/0 is required to access one entry (record) in the chain at a time, and the entire chain has to be read in order to determine where a "new" record has to be added if necessary. In other words, all requests to process background jobs from all sources are queued on a single (long), disk resident JDS queue that is "anchored" to a special permanent job referred to as JOBO. As a result, the time for executing jobs submitted to the background for execution from TSO users at terminals running as foreground sessions is prolonged. When this occurs, users become frustrated because their job has been accepted for processing, e.g. by the time sharing option (TSO) SUBMIT instruction, but it cannot be located. That is, the user cannot determine the status of his/her job, e.g. by using the TSO STATUS command or the operator's INQUIRY command, prior to the job being given a job number.
Another problem is the delay of demand select job processing, e.g. TSO LOGONs and started tasks (initiator). This is a result of the following:
If, deterministically, a particular number of internal readers is selected to be in the system at all times, then, depending upon the amount of work, size of the system, number of users, etc., that specific number of internal readers may be too many or too few. If it were too few, the "bottleneck" would occur again. If it were too many, those internal readers waiting for work would consume system resources, virtual and real storage and, most importantly, cycles required by the dispatcher to scan the queue of jobs looking for actual work to dispatch. Also, looking to the future, a specific number of internal reader jobs "optimally" selected for todays system could create a "bottleneck" in a future system that was much faster than todays system and capable of handling more workload. This would be especially so if I/0 speeds do not keep pace with improvements in processor speeds.
It is therefore an object of this invention to eliminate the bottleneck caused by jobs being submitted to an operating system component faster than the component can receive them.
It is an object of this invention to dynamically, i.e. automatically and continuously, create and cancel internal reader jobs which are available to process the ever-changing number of input job streams while at the same time avoiding excessive system overhead and thrashing.
An object of this invention is to eliminate the need for operators to manually start or cancel internal reader jobs as the number of data sets to be processed varies over time.